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Subject:  Homeland  Defense:  DOD’s  Aerospace  Control  Alert  Basing  Decision  Was  Informed 
by  Various  Analyses 

Since  the  September  1 1 ,  2001 ,  terrorist  attacks,  the  federal  government  has  undertaken 
extensive  efforts  to  protect  U.S.  airspace.  As  a  part  of  the  federal  government’s  overall 
efforts  to  protect  U.S.  airspace,  the  Department  of  Defense  (DOD)  performs  Operation 
Noble  Eagle,  which  consists  of  several  missions,  including  aerospace  control  alert  (ACA).1 
The  ACA  mission  includes  aerospace  control  forces  arrayed  in  a  rapid  response  posture  to 
conduct  both  air  sovereignty  and  air  defense  operations  against  airborne  threats.2  Among 
other  things,  this  includes  fighter  aircraft  and  trained  personnel  on  alert  24-hours  a  day,  365 
days  a  year,  at  16  alert  basing  locations  across  the  contiguous  United  States  and  one  each 
in  Alaska  and  Hawaii  to  deter,  respond  to,  and  if  necessary  defeat  airborne  threats  over  the 
United  States  and  Canada.  These  fighter  aircraft  and  trained  personnel  can  be  the  last  line 
of  multiple  layers  of  the  air  defense  of  the  United  States. 

Our  prior  work  has  highlighted  improvements  and  challenges  in  the  planning  and 
management  of  the  ACA  operation,  including  challenges  related  to  the  North  American 
Aerospace  Defense  Command’s  (NORAD)  and  DOD’s  ability  to  balance  risks,  costs,  and 
benefits  when  making  decisions  about  the  ACA  operation.  Specifically,  in  January  2009  we 


^OD  formerly  referred  to  the  ACA  operation  as  “air  sovereignty  alert  (ASA).” 

2Air  sovereignty  refers  to  a  nation’s  inherent  right  to  exercise  absolute  control  and  authority  over  the  airspace 
above  its  territory.  Air  defense  refers  to  defensive  measures  designed  to  destroy  attacking  enemy  aircraft  or 
missiles  in  the  atmosphere,  or  to  nullify  or  reduce  the  effectiveness  of  such  attack.  See  Joint  Chiefs  of  Staff,  Joint 
Pub.  1-02,  Department  of  Defense  Dictionary  of  Military  and  Associated  Terms  (Nov.  8,  2010;  as  amended 
through  Nov.15,  2012). 
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reported  on  shortcomings  in  DOD’s  management  approach  to  the  ACA  operation.3  Among 
other  things,  we  reported  that  NORAD  faced  difficulty  determining  the  appropriate  levels  and 
types  of  units,  personnel,  and  aircraft  for  the  ACA  operation.  We  made  five 
recommendations  to  DOD  to  improve  management  of  the  ACA  operation,  including 
conducting  routine  risk  assessments  as  part  of  a  risk-based  management  approach.  DOD 
fully  or  partially  agreed  with  all  our  recommendations.  In  January  2012,  we  reported  that  the 
Air  Force  had  taken  steps  to  implement  one  recommendation  and  partially  implement  the 
other  four  recommendations.4  We  also  reported  on  strengths  and  limitations  of  the  computer 
model  NORAD  used  for  assessing  domestic  air  defense  operations,  and  we  reported  that 
NORAD  had  not  considered  cost  in  its  analysis  of  the  ACA  basing  strategy.  NORAD  and 
DOD  had  taken  some  action  to  improve  their  risk-based  management  approach;  however, 
we  made  seven  additional  recommendations  in  January  2012  for  NORAD  and  DOD  to 
implement  a  more  complete  risk-based  management  approach  that  balances  risk  and  cost 
for  the  ACA  operation.  DOD  agreed  with  one  recommendation,  partially  agreed  with  the 
other  six,  and  highlighted  actions  it  would  take  to  implement  the  intent  of  the 
recommendations. 

In  2012,  based  on  the  desire  to  find  savings  in  the  ACA  operation  that  did  not  impede 
operational  effectiveness,  DOD  determined  that  two  ACA  basing  locations  could  be  taken  off 
24-hour  alert.  The  two  locations  are  Duluth,  Minnesota  and  Langley,  Virginia. 

You  asked  us  to  review  DOD’s  analyses  regarding  its  decision  to  change  the  alert  status  for 
those  two  ACA  locations.  This  report  assesses  (1)  the  analyses  NORAD  and  DOD 
conducted  in  support  of  the  decision  to  change  the  alert  status  at  two  NORAD  ACA  basing 
locations,  and  (2)  the  extent  to  which  DOD  estimated  costs  or  cost  savings  resulting  from  its 
decision. 

Scope  and  Methodology 

To  assess  NORAD’s  and  DOD’s  analyses  used  in  their  decision  to  change  the  alert  status  at 
two  NORAD  ACA  basing  locations,  we  reviewed  and  analyzed  applicable  documentation 
and  interviewed  relevant  officials  at  NORAD,  the  Office  of  the  Assistant  Secretary  of 
Homeland  Defense  and  Americas’  Security  Affairs,  the  Office  of  the  Secretary  of  Defense 
Office  of  Cost  Assessment  and  Program  Evaluation  (CAPE),  the  U.S.  Air  Force,  the  National 
Guard  Bureau,  and  the  Air  National  Guard.  We  also  reviewed  prior  GAO  reports  that 
recommended  NORAD  and  DOD  use  a  risk-based  management  approach  to  the  ACA 
operation.5  Drawing  from  our  previous  assessments  to  understand  NORAD’s  ACA  model, 
we  interviewed  NORAD  officials  who  were  familiar  with  the  model  to  update  our 
understanding  of  the  model’s  current  assumptions  and  structure  and  reviewed  its  results 
under  various  scenarios.  We  reviewed  changes  or  improvements  to  the  model  and  risk 
assessment  and  compared  NORAD’s  and  DOD’s  risk  assessment  methodologies  to  GAO 
and  federal  government  best  practices  and  DOD  criteria  for  risk-based  management,  all  of 


3GAO,  Homeland  Defense:  Actions  Needed  to  Improve  Management  of  Air  Sovereignty  Alert  Operations  to 
Protect  U.S.  Airspace,  GAO-09-184  (Washington,  D.C.:  Jan.  27,  2009). 

4GAO,  Homeland  Defense:  Continued  Actions  Needed  to  Improve  Management  of  Air  Sovereignty  Alert 
Operations,  GAO-12-311  (Washington,  D.C.:  Jan.  31, 2012). 

5GAO-09-184  and  GAO-12-311. 
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which  remain  relevant  and  are  described  in  our  previous  reports.6  In  addition,  we  assessed 
the  assumptions  and  structure  of  another  computer  simulation  model  created  by  CAPE  to 
assess  the  risk  of  various  ACA  basing  strategies.  We  observed  a  demonstration  of  the 
model  using  multiple  assumptions,  reviewed  its  results  under  the  various  assumptions,  and 
interviewed  the  CAPE  official  who  developed  it.  This  allowed  us  to  understand  how  CAPE’s 
model  assessed  risk  in  its  analyses  of  ACA  basing  locations.  On  the  basis  of  our 
understanding  of  both  computer  models,  the  results  of  NORAD’s  and  DOD’s  analyses,  and 
discussions  with  relevant  officials,  we  assessed  the  reasonableness  and  reliability  of  the 
analyses  that  NORAD  and  DOD  performed  to  change  the  alert  status  at  two  ACA  basing 
locations. 

To  assess  the  extent  to  which  DOD  estimated  the  costs  or  cost  savings  as  part  of  its 
decision  to  reduce  two  ACA  basing  locations  from  24-hour  alert  status,  we  reviewed 
documents  such  as  the  Air  Force’s  budget  exhibit  and  cost-savings  estimate  and 
interviewed  NORAD,  the  Air  Force,  and  other  DOD  officials.  To  assess  the  Air  Force’s  cost- 
savings  estimate,  we  used  key  principles  we  derived  from  cost  estimating,  budgeting,  and 
related  guidance.7  We  reviewed  the  Air  Force’s  estimate  and  discussed  with  Air  Force 
officials  the  detailed  documentation;  significant  cost  assumptions  and  other  key 
assumptions;  reliability  of  the  data;  and  the  accuracy,  Air  Force  review  of,  and  Air  Force 
validation  of  its  cost-saving  estimate. 

We  conducted  this  performance  audit  from  August  2012  through  February  2013  in 
accordance  with  generally  accepted  government  auditing  standards.  Those  standards 
require  that  we  plan  and  perform  the  audit  and  obtain  sufficient,  appropriate  evidence  to 
provide  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our  audit  objectives. 
We  believe  that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives. 

Results  in  Brief 

DOD’s  decision  to  change  the  alert  status  at  two  ACA  basing  locations  was  informed  by 
various  analyses,  which  assessed  the  impact  on  operational  effectiveness  to  the  ACA 
operation.  DOD’s  analyses  were  based  on  a  NORAD  assessment — which  included  a 
computer  model — a  CAPE  computer  model,  and  an  analysis  by  NORAD’s  Continental  U.S. 
NORAD  Region.  NORAD’s  analyses,  informed  by  a  model  developed  in  response  to  a 
recommendation  in  our  2009  report  and  bolstered  by  additional  NORAD  analysis,  identified 
two  basing  locations  that  could  be  removed  from  24-hour  alert  status  with  little  impact  on 
ACA  capabilities  overall.  In  our  January  2012  report,  we  noted  limitations  to  NORAD’s 


6GAO-09-184  and  GAO-12-311. 

7We  reviewed  federal  guidance  documents  related  to  cost  estimating,  accounting  standards,  economic  analysis, 
and  budgeting  and  identified  broad  themes  that  we  believe  can  be  applied  to  evaluating  cost  analyses  and 
estimates,  including  those  of  cost  savings.  The  guidance  documents  we  reviewed  include:  GAO  Cost  Estimating 
and  Assessment  Guide ,  GAO-09-3SP;  Office  of  Management  and  Budget  Circular  No.  A-11,  Preparation, 
Submission  and  Execution  of  the  Budget  (Aug.  2011,  superseded  by  an  August  2012  issuance);  Federal 
Accounting  Standards  Advisory  Board,  Statement  of  Federal  Financial  Accounting  Standards  4  (June  2011); 
Department  of  Defense  Instruction  7041.3,  Economic  Analysis  for  Decisionmaking  (Nov.  7,  1995);  and 
Department  of  Defense  Financial  Management  Regulation  7000. 14-R,  Volume  4,  Chapter  22,  Cost  Funding 
(May  2010).  Although  each  of  these  documents  may  not  apply  to  these  circumstances  as  a  legal  matter,  we 
believe  that  these  documents  collectively  contain  broad  themes  that  can  be  applied  to  evaluating  cost  analyses. 
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computer  model.  For  example,  we  found  that  it  did  not  include  a  prioritized  list  of 
metropolitan  areas  and  critical  infrastructure  locations  that  NORAD  should  protect  and  that  it 
did  not  incorporate  assumptions  associated  with  all  three  elements  of  risk:  threat, 
vulnerability,  and  consequence.  Since  our  January  2012  report,  NORAD  has  strengthened 
its  risk-based  management  approach  of  the  ACA  operation  by  improving  its  risk  analyses, 
including  to  change  some  of  the  assumptions  used  to  address  vulnerability  and 
consequence  in  its  model.  With  regard  to  the  CAPE  model,  the  CAPE  office  separately 
identified  a  point  below  which  the  number  of  ACA  basing  locations  on  24-hour  alert  could 
not  be  further  reduced  without  materially  increasing  risk.  Both  NORAD’s  and  CAPE’s 
analyses  identified  the  Duluth,  Minnesota  and  Langley,  Virginia  ACA  locations  as  the  best 
candidates  to  take  off  24-hour  alert  status.  In  addition  to  these  two  models,  the  Continental 
U.S.  NORAD  Region  convened  a  panel  of  subject  matter  experts  to  discuss  and  analyze  the 
ACA  basing  strategy.  These  experts’  analysis  and  conclusions  were  consistent  with  the 
results  of  the  analysis  using  NORAD’s  model.  The  Deputy’s  Management  Action  Group — 
which  includes  DOD  senior  leaders  who  monitor  DOD’s  efforts  to  improve  its  defense 
business  operations — approved  the  decision  to  change  the  ACA  alert  status  at  the  two 
basing  locations. 

The  Air  Force  estimated  cost  savings  from  reducing  the  alert  status  at  the  two  ACA  basing 
locations  of  about  $73  million  over  fiscal  years  2013  through  2017.  In  our  January  2012 
report,  we  noted  that  NORAD  had  not  identified  potential  cost  savings  that  could  result  from 
eliminating  a  given  number  of  ACA  basing  locations  on  24-hour  alert.  After  DOD  made  the 
decision  to  remove  the  Duluth  and  Langley  ACA  basing  locations  from  24-hour  alert,  the  Air 
Force  calculated  estimated  cost  savings.  Air  Force  officials  told  us  that  the  estimated  cost 
savings  are  mostly  from  personnel  who  will  no  longer  be  required  to  fully  maintain  24-hour 
alert  status — by  shifting  from  full-time  to  part-time  status.  We  did  not  independently  verify  the 
Air  Force’s  estimated  cost  savings.  However,  we  discussed  with  Air  Force  officials  their  level 
of  review  and  validation  of  the  cost-savings  estimate  and  determined  that  the  Air  Force’s 
categories  of  savings  were  consistent  with  key  principles  of  cost-savings  estimates.  GAO  is 
not  making  recommendations  in  this  report. 

Background 

As  a  component  of  Operation  Noble  Eagle,8  the  ACA  operation  is  conducted  by  several 
DOD  components  under  the  command  of  NORAD  for  the  airspace  over  North  America, 
which  includes  Alaska  and  the  contiguous  United  States,  and  under  the  command  of  U.S. 
Pacific  Command  for  the  airspace  over  the  state  of  Hawaii  and  U.S.  territories  in  the  Pacific. 
The  Joint  Chiefs  of  Staff  have  directed  the  Air  Force  (among  other  services)  to  provide 
support  for  the  ACA  operation.  Generally,  Air  National  Guard  aircraft  and  personnel  are 
used  for  alert,  consistent  with  the  Air  National  Guard’s  homeland  defense  mission.  Both  the 
Air  Force’s  Air  Combat  Command  and  the  National  Guard  Bureau  believe  that  Air  National 
Guard  units  can  conduct  alert  duty  with  less  effect  on  overseas  missions  and  at  a  lower  cost 


8Operation  Noble  Eagle,  which  was  initiated  after  the  terrorist  attacks  of  September  1 1 , 2001 ,  is  DOD’s  umbrella 
homeland  defense  mission  for  North  America  and  Hawaii.  There  are  a  variety  of  operations  that  could  be 
conducted  underneath  this  umbrella  mission.  For  example,  NORAD  could  conduct  irregular  air  patrols  above 
major  metropolitan  areas  and  critical  infrastructure  facilities  (as  opposed  to  more  regular — and  predictable — 
patrols)  as  well  as  maintain  an  alert  force  of  fighter,  tanker,  and  command  &  control  aircraft.  Aircraft  and  alert 
commitments  are  based  on  a  tiered  response  system.  As  threat  levels  intensify,  the  number  of  aircraft  on  alert 
and  on  patrol  increases. 
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than  active-duty  Air  Force  units.9  The  units  at  all  ACA  locations  (i.e. ,  both  Air  National  Guard 
and  active-duty  personnel)  are  dual-tasked  to  conduct  expeditionary  missions,  such  as 
Operation  Enduring  Freedom  in  Afghanistan,  as  well  as  the  ACA  operation. 

The  ACA  operation  consists  of  ground  operations  that  take  place  before  fighter  aircraft  take 
off,  including  such  activities  as  maintaining  the  fighter  aircraft.  This  operation  also  includes 
those  activities  that  may  take  place  after  a  unit  receives  an  alert  from  NORAD  but  before  the 
aircraft  are  airborne.  For  example,  pilots  and  maintenance  personnel  may  rush  from  their 
nearby  lodging  facility  to  the  alert  aircraft  facility,  where  maintenance  personnel  conduct  final 
preparations  while  the  pilots  sit  in  their  aircraft  awaiting  further  instruction  (battle  stations). 
Alternatively,  pilots  may  taxi  the  aircraft  to  the  end  of  the  runway  and  await  further 
instruction  (runway  alert)  or  take  off  in  response  to  the  alert  (scramble).  Once  aircraft  take 
off,  Air  National  Guard  pilots  in  Title  32  status  under  the  command  and  control  of  the  state 
governor,  automatically  convert  to  federal  Title  10  status  under  the  command  and  control  of 
NORAD.10  Within  limits,  NORAD  can  increase  personnel,  aircraft,  and  the  number  of  ACA 
basing  locations  on  the  basis  of  changes  in  the  threat  conditions. 

In  response  to  one  of  the  recommendations  in  our  2009  report — to  conduct  routine  risk 
assessments  to  determine  ACA  requirements,  including  the  appropriate  numbers  of  ACA 
basing  locations,  personnel,  and  aircraft  to  support  the  ACA  operation — NORAD  officials 
developed  a  quantitative  computer  simulation  model  to  study  basing  locations  that  were  part 
of  the  ACA  operation.11  NORAD’s  computer  model  provides  DOD  with  data  to  assess  risk  of 
the  ACA  operation,  which  helps  decision  makers  identify  and  evaluate  potential  risks 
associated  with  various  allocations  of  resources.  The  model  allows  NORAD  to  analyze  how 
changing  the  number  and  location  of  ACA  basing  locations  would  affect  its  ability  to  defend 
the  homeland  from  commercial  aviation  attacks.  The  model  uses  real-world  data  on  the 
opportunity  for  attacks  from  within  the  commercial  aviation  system  and  on  NORAD’s 
operational  capabilities,  along  with  assumptions  about  the  randomness  of  these  factors  in 
practice,  to  generate  numerous  simulated  Operation  Noble  Eagle-related  events  and  assess 
NORAD’s  capability  to  defeat  them.12  By  assessing  ACA  operational  capabilities  for  various 
combinations  of  basing  locations,  the  model  helps  NORAD  quantify  how  risk  would  change 
if  certain  locations  were  removed  from  24-hour  alert  status. 

In  2010,  NORAD  used  the  model  to  analyze  the  pre-existing  16  ACA  basing  locations  in  the 
contiguous  United  States,  in  order  to  identify  the  potential  operational  performance  and 
efficiencies  of  alternative  ACA  basing  locations.  According  to  NORAD  officials,  the  results  of 


9The  National  Guard  Bureau  is  a  joint  function  within  DOD  that,  among  other  things,  participates  with  staff  from 
the  Department  of  the  Army  and  the  Department  of  the  Air  Force  in  the  formulation,  development,  and 
coordination  of  all  programs,  policies,  concepts,  and  plans  pertaining  to  or  affecting  the  National  Guard,  the  Army 
National  Guard  of  the  United  States,  and  the  Air  National  Guard  of  the  United  States.  The  Air  National  Guard  of 
the  United  States  is  a  reserve  component  of  the  U.S.  Air  Force. 

i  n 

According  to  Air  National  Guard  Instruction  1 0-203:  Air  National  Guard  (ANG)  Alert  Resource  Management 
(February  22,  2012),  provisions  will  be  made  to  ensure  members  automatically  convert  to  Title  10,  if  a  specific 
Title  10  trigger  is  included  in  higher  headquarters  guidance.  Title  32  and  Title  10  refer  to  sections  of  the  United 
States  Code. 

^Referred  to  in  NORAD  documents  as  the  Operation  Noble  Eagle  Fighter  Basing  Location  Model. 

12The  algorithms  that  calculate  different  scenarios  or  events  to  determine  probability  of  certain  outcomes  are 
referred  to  as  Monte  Carlo  simulations. 
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this  analysis  led  DOD  officials  to  direct  NORAD  to  determine  which  locations,  if  any,  might 
be  removed  from  24-hour  alert.  In  201 1 ,  after  DOD  began  to  plan  for  the  reduction  of  ACA 
basing  locations  on  24-hour  alert,  NORAD  performed  additional  analyses,  using  both  the 
NORAD  model  and  a  less  complex  calculation,  to  determine  the  specific  ACA  basing 
locations  that  could  be  taken  off  24-hour  alert.  The  analyses  sought  to  identify  the  two 
locations  where  alert  status  could  be  reduced  with  the  smallest  impact  on  NORAD’s  ACA 
capabilities. 

In  our  prior  reports,13  we  stated  that,  as  part  of  an  overall  risk-based  management 
framework,  routine  risk  assessments — which  the  NORAD  model  and  related  analyses  help 
to  address — should  contain  three  key  elements:  an  analysis  of  threat,  an  estimation  of 
vulnerability,  and  identification  of  consequences.  In  our  2012  report,  we  noted  that  NORAD’s 
model  was  limited  because  it  was  designed  to  assess  the  ability  of  its  aircraft  to  respond  to  a 
single  type  of  threat  regardless  of  its  location  in  the  United  States  and  that  it  did  not  include 
a  prioritized  list  of  metropolitan  areas  and  critical  infrastructure  NORAD  should  protect. 

DOD’s  ACA  Decision  Was  Informed  by  Various  Analyses  that  Produced 
Similar  Conclusions 

DOD’s  decision  to  change  the  alert  status  at  two  ACA  basing  locations  was  informed  by 
various  analyses  that  assessed  the  impact  on  operational  effectiveness  to  the  ACA 
operation.  These  DOD  analyses  were  based  on  a  NORAD  assessment — which  included  a 
computer  model — a  CAPE  computer  model,  and  a  Continental  U.S.  NORAD  Region14 
assessment.  We  assessed  the  analytical  and  methodological  basis  for  the  NORAD  and 
CAPE  computer  simulation  models  and  found  them  to  be  reasonable  based  on  our 
understanding  of  the  structure,  algorithms,  and  outputs  of  these  models.  Although  the 
methodology  used  in  the  three  analyses  differed,  they  reached  similar  conclusions.  Based 
on  the  analytical  conclusions,  the  Deputy’s  Management  Action  Group — which  includes 
DOD  senior  leaders  who  monitor  DOD’s  efforts  to  improve  its  defense  business 
operations — approved  the  decision  to  change  the  ACA  alert  status  at  the  two  basing 
locations. 

DOD  used  the  various  analyses  by  NORAD,  CAPE,  and  the  Continental  U.S.  NORAD 
Region,  which  produced  similar  results,  to  inform  its  decision  to  reduce  the  alert  status  at 
Duluth,  Minnesota  and  Langley,  Virginia.  NORAD’s  analysis  used  criteria,  such  as  historical 
data  on  the  number  of  actual  responses  by  each  ACA  basing  location,  and  was  informed  by 
NORAD’s  computer  model.  NORAD  officials  told  us  that  its  model’s  assumptions  about 
threat  had  not  changed  since  our  last  report  and  that  the  model  could  not  be  used  to 
consider  some  types  of  airborne  threats.  Therefore,  NORAD  used  additional  calculations  of 
such  things  as  basic  fighter  response  times  to  help  inform  its  broader  analysis.  We  found 
that  NORAD  improved  its  risk-based  management  approach  of  the  ACA  operation  by 
improving  its  risk  analyses,  specifically  by  changing  some  of  the  assumptions  used  to 
address  vulnerability  and  consequence.  For  example,  NORAD  officials  stated  that,  although 
the  list  of  metropolitan  areas  and  critical  infrastructure  had  not  been  updated  since  DOD 


13GAO-12-31 1  and  GAO-09-184. 

14A  component  of  NORAD,  the  Continental  U.S.  NORAD  Region  provides  airspace  surveillance  and  control  and 
directs  aerospace  control  activities  for  the  contiguous  United  States. 
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developed  it  after  the  September  1 1 ,  2001 ,  attacks,  NORAD  prioritized  the  assets  to  protect, 
and  its  model  used  those  priorities  in  its  current  analyses  of  ACA  basing  locations.  NORAD’s 
analysis  indicated  that  two  basing  locations  could  be  removed  from  24-hour  alert  status  with 
little  impact  on  ACA  capabilities  overall. 

We  found  that  DOD’s  CAPE  office  developed  its  own  computer  model  that  could  analyze  the 
basing  locations  of  Operation  Noble  Eagle,  specifically,  the  ACA  operation.  CAPE’s  model 
calculated  population  and  infrastructure  coverage  for  all  combinations  of  ACA  basing 
locations.  The  model  used  various  assumptions  about  NORAD’s  operational  capabilities — 
such  as  flight  times  and  aircraft  velocity — with  a  goal  of  optimizing  the  number  and 
placement  of  the  ACA  basing  locations  that  existed  in  the  United  States.  In  contrast  to 
NORAD’s  model,  CAPE’s  analyses  made  no  assumptions  about  where  threats  would 
emerge  and  calculated  NORAD’s  ACA  capabilities  at  any  geographic  point.  CAPE’s  results 
identified  a  point  below  which  the  number  of  ACA  basing  locations  on  24-hour  alert  could 
not  be  further  reduced  without  materially  increasing  risk.  CAPE  officials  stated  that  their 
model  could  analyze  numerous  alternative  ACA  basing  location  scenarios  more  quickly  than 
NORAD’s  model.  Although  CAPE’s  model  was  developed  independently  of  NORAD,  both 
CAPE  and  NORAD  officials  told  us  that  they  discussed  the  CAPE  model’s  assumptions  with 
one  another  to  ensure  their  accuracy  in  such  areas  as  realistic  fighter  scramble  times  and 
operational  range. 

In  addition  to  its  quantitative  analysis,  NORAD  asked  its  Continental  U.S.  NORAD  Region  to 
conduct  a  separate  analysis  regarding  the  operational  efficiencies  and  capabilities  of  the 
ACA  operation.  This  analysis  relied  on  Continental  U.S.  NORAD  Region  subject  matter 
experts  on  strategy,  planning,  logistics,  and  tactics  for  the  ACA  operation,  rather  than  a 
computer  simulation  model.  According  to  these  experts,  they  used  criteria  such  as  time, 
distance,  major  population  areas,  and  proximity  to  international  flight  routes  in  measuring 
the  overlapping  coverage  of  select  ACA  basing  locations,  considering  various  terrorist  threat 
scenarios.  These  experts’  analysis  and  conclusion  were  consistent  with  the  results  of 
NORAD  headquarters’  analysis.  NORAD  officials  told  us  that  the  results  of  the  NORAD  and 
Continental  U.S.  NORAD  Region  analyses  were  approved  by  the  NORAD  Commander  and 
provided  in  a  briefing  to  congressional  staff. 

Although  the  specific  methods  used  in  the  various  analyses  of  the  ACA  operation  differed, 
we  found  that  NORAD’s  and  DOD’s  use  of  various  analytical  methods  comprised  a 
reasonable  approach  to  evaluating  changes  in  ACA  basing  locations.  The  NORAD  model 
and  response  time  analysis,  the  Continental  U.S.  NORAD  Region  analysis,  and  the  CAPE’s 
model  used  different  methods  to  evaluate  the  ACA  operation  and  the  risk  of  various  ACA 
basing  options.  For  example,  the  model  that  contributed  to  NORAD’s  analysis  uses  real- 
world  data  on  the  opportunity  for  attacks  from  within  the  commercial  aviation  system  and  on 
NORAD’s  operational  capabilities,  along  with  assumptions  about  the  randomness  of  these 
factors  in  practice,  and  it  estimated  how  response  times  would  change  if  various  ACA 
basing  locations  were  taken  off  24-hour  alert.  NORAD  used  various  criteria,  including 
historical  data  on  ACA  basing  location  responses,  to  determine  which  ACA  locations  could 
be  taken  off  24-hour  alert.  The  Continental  U.S.  NORAD  Region  convened  a  panel  of 
subject  matter  experts  to  analyze  the  ACA  operation.  In  contrast  to  NORAD’s  model, 

CAPE’s  model  made  no  assumptions  about  where  threats  would  emerge,  and  calculated 
NORAD’s  ACA  capabilities  to  reach  maximal  populations  at  any  geographic  point.  Each 
analysis  concluded  that  taking  the  Duluth,  Minnesota,  and  Langley,  Virginia,  ACA  locations 
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off  24-hour  alert  status  would  result  in  the  smallest  impact  on  ACA  capabilities  and  not 
materially  increase  risk. 


The  results  of  CAPE’s  independent  model-driven  analyses  regarding  the  reduction  of  ACA 
basing  locations  on  24-hour  alert  was  brought  to  the  Deputy’s  Management  Action  Group, 
which  includes  DOD  senior  leaders,  led  by  the  Deputy  Secretary  of  Defense,  who  monitor 
DOD’s  efforts  to  improve  its  defense  business  operations.  CAPE  officials  stated  that  after 
discussing  the  ACA  basing  location  options,  the  Deputy’s  Management  Action  Group 
approved  the  decision  to  change  the  ACA  alert  status  at  the  two  ACA  basing  locations — 
Duluth,  Minnesota  and  Langley,  Virginia. 

The  Air  Force  Estimated  Cost  Savings  to  the  ACA  Operation 

NORAD  and  Air  Force  officials  told  us  that  the  Air  Force  estimated  cost  savings  of  $73 
million  over  five  years  after  DOD  made  the  final  decision  to  reduce  the  alert  status  at  the  two 
ACA  basing  locations,  as  part  of  the  Secretary  of  Defense’s  efficiency  initiative,15  not  as  an 
effort  to  conduct  a  complete  cost-benefit  assessment  of  ACA  basing  options.  In  our  January 
2012  report,  we  noted  that  NORAD’s  2010  analysis  did  not  identify  potential  cost  savings 
that  could  result  from  eliminating  a  given  number  of  ACA  basing  locations.  As  noted  in  that 
report,  an  analysis  that  balances  both  risk  and  costs  would  enable  NORAD,  DOD,  and 
Congress  to  make  more  fully-informed  decisions  about  the  number  and  location  of  ACA 
bases.  According  to  Air  Force  officials,  the  cost-savings  estimate  was  calculated  for  the 
fiscal  year  2013  budget  submission  and  included  annual  cost-savings  estimates  that  ranged 
from  about  $9.2  million  to  $16.6  million  for  fiscal  years  2013  through  2017,  as  shown  in 
Table  1. 


Table  1 :  Estimated  Cost  Savings  of  the  ACA  Operation  for  Fiscal 
Years  2013 -2017 

(in  millions) 

Fiscal  Year 

Savings 

2013 

$9.2 

2014 

15.4 

2015 

15.8 

2016 

16.1 

2017 

16.6 

Total 

$73.1 

Source:  GAO  analysis  of  DOD  data. 


Air  Force  officials  told  us  that  the  majority  of  the  cost  savings  were  from  personnel  who 
would  shift  from  full-time  to  part-time  status.  Other  areas  from  which  the  Air  Force  identified 
savings  included  decreases  to  operations  and  maintenance  costs  at  the  basing  locations. 


15ln  May  2010,  the  Secretary  of  Defense  publicly  announced  that  DOD  was  to  undertake  a  department-wide 
initiative  to  assess  how  the  department  is  staffed,  organized,  and  operated  with  the  goal  of  reducing  costs  and 
modernizing,  among  others,  its  weapons  portfolio.  The  Secretary’s  initiative  targeted  both  short-  and  long-term 
improvements  and  set  specific  goals  and  targets  for  achieving  cost  savings  and  efficiencies,  which  are  expected 
to  be  achieved  between  fiscal  years  2012  and  2016. 


Page  8 


GAO-1 3-230R  Homeland  Defense 


We  did  not  independently  verify  the  Air  Force’s  estimated  cost  savings.  However,  we 
compared  the  Air  Force’s  estimate  to  key  principles  we  derived  from  cost  estimating, 
budgeting,  and  related  guidance.  For  example,  a  cost-savings  estimate  should  have  an 
appropriate  level  of  detailed  documentation,  all  significant  costs  and  key  assumptions 
should  be  included,  the  reliability  of  the  data  to  develop  the  estimate  should  be  assessed, 
and  the  accuracy  of  the  calculations  should  be  verified  or  validated.  According  to  Air  Force 
officials,  the  cost-savings  estimate  was  reviewed  and  verified  by  knowledgeable  high-level 
Air  Force,  National  Guard  Bureau,  and  CAPE  officials  to  ensure  that  costs  were  valid,  key 
assumptions  were  included,  and  the  calculations  were  accurate  and  reliable.  Therefore,  we 
determined  that  the  Air  Force’s  categories  of  savings  were  consistent  with  the  key  principles 
of  preparing  cost-savings  estimates. 

Agency  Comments 

NORAD  and  DOD  reviewed  a  draft  of  this  report  and  provided  technical  comments,  which 
we  incorporated  as  appropriate. 


We  are  sending  copies  of  this  report  to  appropriate  congressional  committees,  the  Secretary 
of  Defense,  the  Commander  of  NORAD,  the  Secretary  of  the  Air  Force,  and  the  Chief  of  the 
National  Guard  Bureau.  This  report  also  is  available  at  no  charge  on  our  Web  site  at 

http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  about  this  report,  please  contact  me  at  (202)  51 2- 
6912  or  gamblerr@gao.gov.  Contact  points  for  our  Offices  of  Congressional  Relations  and 
Public  Affairs  may  be  found  on  the  last  page  of  this  report.  GAO  staff  who  made  major 
contributions  to  this  report  are  listed  in  the  enclosure. 


Rebecca  Gambler 
Director 

Defense  Capabilities  and  Management 


Enclosure 


Page  9 


GAO-1 3-230R  Homeland  Defense 


Enclosure:  GAO  Contact  and  Staff  Acknowledgments 


GAO  Contact 

Rebecca  Gambler,  (202)  512-6912  or  gamblerr@gao.gov 

Staff  Acknowledgments 

In  addition  to  the  contact  named  above,  Joseph  Kirschbaum,  Assistant  Director;  Pat  L 
Bohan;  Susan  Ditto;  Gregory  Marchand;  Charles  Perdue;  Carol  Petersen;  Amie  Steele;  Jeff 
Tessin;  and  Ed  Yuen  made  significant  contributions  to  the  report. 


(351767) 


Page  10 


GAO-1 3-230R  Homeland  Defense 
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